Skip to content

cache metrics: route revalidation, origin duration, Cache-Control - #4386

Closed
larry-dalmeida wants to merge 5 commits into
zalando:larry-almeida-zse-cache-timeouts-prfrom
larry-dalmeida:metrics-cde-origin-control
Closed

larry-dalmeida wants to merge 5 commits into
zalando:larry-almeida-zse-cache-timeouts-prfrom
larry-dalmeida:metrics-cde-origin-control

Conversation

@larry-dalmeida

@larry-dalmeida larry-dalmeida commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary of changes

Completes metrics with route revalidation outcomes, origin fetch duration, and Cache-Control directive tracking.

How to Verify

Shared setup

mkdir skipper-review && cd skipper-review
git clone https://github.com/zalando/skipper.git skipper && cd skipper
for PR in 4369 4370 4371 4372 4384 4385 4386 4387; do
  git fetch origin pull/$PR/head:pr-$PR
  git worktree add ../skipper-pr-$PR pr-$PR
done
cd ..
git clone https://github.com/larry-dalmeida/skipper-cache-e2e-tests.git
cd skipper-cache-e2e-tests
make verify SKIPPER_CHECKOUT=../skipper-pr-4387

make verify starts or reuses Minikube, the mock origin and the stack; runs the smoke and coalescing checks; and validates supported dashboard queries. It prints a report path and exits non-zero on failure. For outage evidence, run make chaos SKIPPER_CHECKOUT=../skipper-pr-4387 NAME=origin-stale; it includes its own timed load. Requirements: Python 3.9+, Make, Bash, Git, curl, kubectl, Minikube, Docker and Go compatible with Skipper's go.mod. Free ports: 8080, 8081, 6379, 4318, 9091, 9999, 9911 (8089 for endless load). Dynamic URLs: .runtime/urls.txt; stop with make stop.

Validated on #4387 (d767f636): smoke: 6,398 requests, 0 failures; origin-stale: 155 failed revalidations across 77 routes. Smoke metric checks: 42 PASS, 2 fault-only NOT EXERCISED, 0 FAIL.

Focused tests

go -C ../skipper-pr-4386 test -race -count=1 ./filters/cache -run '^(TestCacheOriginStatusClass|TestCacheRecordCacheControlBoundedLabels|TestCacheRecordCacheControlOncePerResponse|TestCacheFilterOriginMetrics|TestCacheRouteRevalDropped|TestCacheResponseOriginCacheControl|TestCacheRouteRevalFailurePaths|TestCacheRouteRevalTerminalOutcomes)$'
go -C ../skipper-pr-4386 test -race -count=1 ./metrics -run '^(TestPrometheusCacheRouteReval|TestPrometheusCacheOriginDurationRouteStatus|TestPrometheusCacheOriginCacheControl)$'
go -C ../skipper-pr-4386 test -race -count=1 ./metrics/metricstest -run '^TestCacheOriginMockMetrics$'

Expected: 12 focused tests pass; route-revalidation, origin-duration and bounded Cache-Control metrics emit samples. During origin-stale, origin 5xx increments failed (including cached errorTTL responses); legacy cache.reval_error now counts 5xx too and counts body-read failures once. Smoke alone does not exercise failures; see the shared outage report.

Checklist

Rollback Plan

Revert this PR to restore previous metrics behavior.

References & Additional Context

Depends on #4384 and #4385. Completes the metrics stack with route revalidation, origin latency, and Cache-Control directive metrics.

E2E Evidence

Latest-chain verification: origin-stale report records 155 failed revalidations across 77 routes; smoke report has 0 failures. The stack run uses #4387, which includes this PR.

@larry-dalmeida
larry-dalmeida force-pushed the metrics-cde-origin-control branch from 696a8d3 to bda674b Compare October 26, 2026 13:25
@larry-dalmeida larry-dalmeida changed the title [cache metrics C/D/E] family reval, origin duration, cache-control cache metrics: family reval, origin duration, cache-control Oct 26, 2026
Add skipper_cache_lru_entries{route} and skipper_cache_lru_capacity_bytes gauges. ShardedByteLRU.Usage returns bytes and entry count together; Bytes delegates to it.

Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>
Count hit, stale, and miss outcomes per route ID and record the time from process start to the first cache hit.

Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>
- Route revalidation outcomes (started/success/failed/dropped)
- Origin fetch duration by route and status class
- Cache-Control directive tracking from origin responses

These metrics instrument cache operations across routes and origin interactions.

Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>
Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>
Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>
@larry-dalmeida
larry-dalmeida force-pushed the metrics-cde-origin-control branch from 78fb3ac to f9a04f6 Compare October 27, 2026 07:50
@larry-dalmeida larry-dalmeida changed the title cache metrics: family reval, origin duration, cache-control cache metrics: route revalidation, origin duration, Cache-Control Oct 27, 2026
@larry-dalmeida

Copy link
Copy Markdown
Collaborator Author

Superseded by #4395, stacked above #4394. Changes consolidated with published metric, dashboard and fault-test evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant